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Abstract 

One  of  the  most  important  tasks  in  the  systems 
development  process  is  that  of  performance 
analysis.  It  is  needed  to  ensure  that  the  system 
meets  its  requirements,  is  delivered  on  schedule, 
and  developed  within  allocated  costs.  It  consists  of 
two  phases:  performance  prediction  and 
performance  measurement.  Proper  selection  of 
performance  measurement  attributes  is  essential  to 
this  process.  These  measurement  attributes 
commonly  called  “measures  of  effectiveness”  or 
“MOEs”  provide  quantifiable  benchmarks  against 
which  the  system  concept  and  implementation  can 
be  compared.  Early  in  the  life  of  a  system, 
prediction  is  required  for  feasibility  and 
specification  development.  Towards  the  end  of 
systems  implementation  and  development, 
performance  measurement  techniques  play  a  major 
role  in  system  testing  and  verification.  Choosing 
incorrect  MOEs  will  result  in  a  system  that  does 
not  meet  customer  expectations.  This  paper 
introduces  a  comprehensive  and  systematic 
process  by  which  viable  MOEs  that  quantify  and 
analyze  system  performance  may  be  developed. 
The  approach  is  based  on  research  into  Command 
and  Control  System  evaluation  performed  at  the 
Naval  Postgraduate  School  during  the  late  1980’s. 
This  paper  extends  the  research  to  open  systems  in 
general  and  develops  several  of  the  original 
theoretical  concepts  in  more  detail. 

Introduction 

The  design  of  a  system  is  an  ill-posed  problem  that 
has  no  solution  without  a  set  of  criteria  to  guide 
choices  [Oliver  et.  al.,  1997].  Morse  and  Kimball 
first  addressed  the  issue  of  performance  prediction 
and  measurement  in  the  summary  of  their  World 
War  II  analytic  work  published  as  “Methods  of 
Operations  Research”  [Morse  and  Kimball,  1970].* 
They  cite  an  excellent  example  of  how  the  role  of 
effectiveness  measures  in  systems  thinking  can  be 
used  to  shape  systems  to  operate  in  a  particular 
environment.  Antisubmarine  warfare  systems 
were  a  high  priority  because  of  the  U-boat  threat 


*  The  work  was  originally  known  as  Operations 
Evaluation  Group  Report  (OEG)  58. 


and  the  attrition  of  merchant  shipping.  The 
question  to  be  answered  was  what  was  the 
difference  between  an  aggressive  and  defensive 
use  of  antisubmarine  formations  and  their 
component  antisubmarine  warfare  systems?  For 
the  former  use,  the  expected  number  of  U-boats 
killed  by  an  antisubmarine  hunter-killer  group  was 
the  effectiveness  measure.  For  the  latter,  the 
effectiveness  measure  was  the  probability  of 
preventing  convoy  formation  penetration  by  the  U- 
boat.  Thus,  while  both  formations  had  ships  as 
basic  component  systems,  hunter-killer  groups 
were  centered  on  an  aircraft  carrier  and 
antisubmarine  destroyers  employing  a  methodical 
search  and  destroy  process.  The  convoy  was 
comprised  of  merchant  ships  and  destroyer  escorts 
using  high-speed  transit  of  the  submarine  areas  as 
a  defensive  process.  Thus,  it  can  be  seen  from  this 
example  that  effectiveness  measures  are  critically 
important  because  they  are  the  criteria  that  drive 
the  system  solution  that  is  found. 

Selection  of  Performance  Measure  Attributes 

Oliver’s  approach  to  bringing  definition  to  the  ill- 
defined  problem  is  to  break  the  systems 
engineering  process  into  two  parts:  what  he 
describes  as  the  Systems  Engineering  Management 
process  and  the  Systems  Engineering  Technical 
process.  Within  the  Systems  Engineering  core 
technical  process  he  describes  six  steps  as  shown 
in  Figure  1.  This  process  recognizes  that  an  open 
system  interacts  with  the  environment  beyond  its 
boundaries.  Steps  2,  3,  and  4  are  crucial  in  that 
they  bound  the  system,  capture  the  systems 
behavior  and  define  the  effectiveness  measures, 
the  criteria  that  mean  success  or  failure.  His 
approach  is  based  upon  the  concept  that  a  system 
is  a  unified  collection  or  set  of  objects  that  exhibit 
a  unique  set  of  behaviors  when  combined  together 
in  their  operating  environment.  It  is  very  similar 
to  the  approach  shown  in  Figure  2.  This  approach 
was  developed  by  the  Military  Operation  Research 
Society’s  (MORS)  work  on  Measures  of 
Effectiveness  for  Command  and  Control.  It  also 
focuses  on  an  early  bounding  of  the  system 
followed  by  selection  of  performance  measures 
[Sweet,  et.  al.,  1985]. 
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The  logic  behind  bounding  the  system  early  is 
driven  by  the  performance  measures  development 
process.  The  system  boundaries  define  the  set  of 
system  parameters  that  drive  system  performance. 
A  change  in  the  boundaries  changes  the  parameter 
set  and  the  resulting  system  behavior  and  its 
performance.  This  is  often  overlooked  in  the 


system  development  process.  There  is  an 
expectation  that  there  is  a  magic  list  of  canned 
effectiveness  measures  that  the  Systems  Engineers 
can  use  like  a  lookup  table  in  the  early  stages  of 
development.  Failure  to  understand  this  point  can 
have  a  ripple  effect  throughout  the  system 
lifecycle. 


Figure  I.  The  Steps  of  the  Core  Technical  Process 


Definition  of  Effectiveness  Measures 

There  are  a  number  of  terms  used  to  describe 
system  performance.  While  several  of  these  terms 
are  often  used  interchangeably  to  describe  the 
same  thing  (e.g.,  Measures  of  Performance 
(MOPs)  is  interchanged  with  Measures  of 
Effectiveness  (MOEs),  the  MORS’s  work 
recognized  that  there  is  indeed  a  hierarchy  of 
effective  measures.  MORS  identified  the 
following  key  concepts:  parameters,  Measure(s)  of 
Performance,  Measure(s)  of  Effectiveness,  and 
Measure(s)  of  Force  Effectiveness.  While  the  later 
term  is  not  appropriate  for  systems  in  general  the 
idea  is  valid  and  will  be  addressed  below. 

This  hierarchy  follows  the  system  of  system 
concept  first  presented  by  Ackoff  [Ackoff,  1971]. 
The  following  definitions  clarify  the  hierarchy. 

Parameters:  the  properties  or  characteristics 
inherent  in  the  physical  entities,  whose  values 
determine  system  behavior  and  the  structure  under 
question,  even  when  not  operating.  Typical 
examples  include  signal-to-noise  ratio,  bandwidth, 
frequency,  aperture  dimensions,  and  bit  error  rates. 


Figure  2.  The  MORS’s  Command  and  Control 
System  Definition  Proeess 
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Measures  of  Performance  (MOP):  measures 
derived  from  the  dimensional  parameters  (both 
physical  and  structural)  and  measure  attributes  of 
system  behavior.  MOPs  quantify  the  set  of 
selected  parameters.  Examples  include  sensor 
detection  probability,  sensor  probability  of  false 
alarm,  and  probability  of  correct  identification. 

Measures  of  Effectiveness  (MOE):  measure  of 
how  a  system  performs  its  functions  within  its 
environment.  An  MOE  is  generally  an 
aggregation  of  MOPs.  Examples  include 
survivability,  probability  of  raid  annihilation,  and 
weapon  system  effectiveness. 

In  the  MORS  work  the  term  Measures  of  Force 
Effectiveness  was  defined  as: 

Measures  of  Force  Effectiveness  (MOFE): 

measure  of  how  a  system  and  the  force  (sensors, 
weapons,  C3  system)  of  which  it  is  a  part  perform 
military  missions. 


determine  to  what  extent  user  requirements  are 
met.  They  are  the  criteria  used  to  make  the  trade¬ 
off  decisions  of  what  to  build.  They  are  the 
criteria  that  drive  the  system  solution  that  is  found. 

Effectiveness  measures  are  derived  from  first 
principles.  They  are  mission  and  scenario 
dependent  and  must  discriminate  between  choices. 
System  parameters  are  often  mistakenly  used  as 
effectiveness  measures.  As  an  example,  sensor 
search  rate  has  been  specified  as  an  MOP, 
however,  increasing  the  search  rate  of  a  sensor 
improves  the  probability  of  detection  thus  search 
rate  is  a  parameter. 

Effectiveness  measures  must  be  both  measurable 
and  testable.  This  means  that  they  are  quantitative 
in  nature.  Further,  they  have  to  realistically 
measure  the  systems  purpose  or  objective.  Failure 
to  do  so  results  in  a  system  that  fails  to  meet  its 
purpose. 


This  last  definition  can  be  modified  for  the  general 
systems  case  as  follows: 

Measures  of  Systems  Effectiveness  (MOSE): 

measure  of  how  a  system  of  systems  performs  its 
mission. 

The  relationship  between  the  various  elements  of 
the  hierarchy  is  shown  in  Figure  3. 


Figure  3.  The  Effectiveness  Measures 
Hierarchy 

Features  of  measures 

With  effectiveness  measures  defined  and  ordered  it 
bears  repeating  that  they  are  the  standards  against 
which  the  performance  of  a  system  is  compared  to 


The  issue  of  sensitivity  is  important.  The 
effectiveness  measure(s)  not  only  needs  to  reflect  a 
change  in  the  parameter  set,  it  must  also  have  a 
reference  from  which  the  change  can  be  evaluated. 
Doubling  the  value  of  a  parameter  does  not 
necessarily  correspond  to  a  doubling  of  the 
effectiveness  measure.  Expressing  MOPs,  MOEs, 
and  MOSEs  as  a  probability  allows  us  to 
determine  if  a  parametric  change  is  statistically 
significant. 

Finally,  effectiveness  measures  must  be 
independent  at  the  level  of  analysis  under 
evaluation.  In  other  words,  MOPs  should  be 
independent  but  they  can  be  aggregated  into 
MOEs.  The  MOEs  would  be  independent  of  each 
other  and  can  be  aggregated  into  an  MOSE. 

Table  1  summarizes  the  desired  characteristics  of 
measures. 


Characteristics 

Definition 

Mission  Oriented 

Relates  to  force/system. 

Discriminatory 

Identifies  real  difference 
between  alternatives. 

Measurable 

Can  be  computed  or 
estimated. 

Qnantitative 

Can  be  assigned  numbers 
or  ranked. 

Realistic 

Relates  realistically  to  the 
system  and  associated 
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uncertainties. 

Objective 

Defined  or  derived, 
independent  of  subjective 
opinion. 

Appropriate 

Relates  to  acceptable 
standards  and  analysis 
objectives. 

Sensitive 

Reflects  changes  in 
system  variables. 

Inclusive 

Reflects  those  standards 
required  by  the  analysis 
objectives. 

Independent 

Mutually  exclusive  with 
respect  to  other  measures. 

Simple 

Easily  understood  by  the 
user. 

Table  I.  Desired  Characteristics  of 
Effectiveness  Measures 

Application 

Figure  4  captures  the  relationship  of  the 
parameters  and  effectiveness  measures  in  the 
performance  prediction  process.  The  parameters 
are  input  into  the  modeling  process  along  with  the 
scenario  requirements  and  environmental 
conditions.  The  effectiveness  measures  are  the 
logical  output. 


Scenario 

Inputs 


Figure  4.  Modeling  system  Performance  [Leite 
and  Mensh,  1999] 

Consider  the  following  example:  a  football  team  is 
a  system  of  systems.  Its  effectiveness  measure  is 


its  probability  of  winning.  It  is  composed  of  four 
major  subsystems,  coaching  staff,  offense,  defense 
and  special  teams.  An  example  MOE  would  be 
the  probability  of  calling  scoring  plays  (or  a 
sequence  of  plays  that  lead  to  a  score)  by  the 
coaching  staff.  A  MOP  would  be  the  probability 
of  completing  a  pass  by  the  offense  or  the 
defense’s  probability  of  causing  an  incomplete 
pass.  The  mix  of  players  on  the  field  would 
determine  the  parameters  (yards  per  carry,  etc.)  at 
any  given  time.  Changing  the  quarterback  would 
change  the  probability  of  completing  a  pass. 

The  Role  of  Time 

Time  is  often  used  as  an  effectiveness  measure. 
The  concept  of  timeliness  is  attractive  as  an 
effectiveness  measure.  However,  this  paper  argues 
that  time  is  a  parameter  and  thus  should  not  be  cast 
in  this  role.  Time  is  the  independent  variable  and 
outcomes  of  processes  occur  with  respect  to  time. 
L’Etoile  defines  Critical  Time  (Tc)  as  the  time 
within  which  the  mission  (task)  must  be  completed 
to  be  successful  [L’Etoile,  1985].  Using  Tc  as  the 
independent  variable,  the  effectiveness  measure  is 
the  probability  of  completing  the  task  within  the 
allowed  time. 


Summary 

The  system  bounding  process  is  the  starting  point 
for  determining  effectiveness  measures  in  that  it 
defines  the  set  of  parameters  and  their  hierarchical 
structure  within  the  system  of  systems  that  drives 
system  performance.  Effectiveness  measures  are 
also  hierarchical  with  MOPs  determined  by  sets  of 
parameters,  MOEs  building  off  the  aggregation  of 
MOPs,  and  MOSEs  building  from  the  MOEs. 
Care  must  be  taken  to  ensure  that  effectiveness 
measures  reflect  the  systems  objective.  Care  must 
also  be  taken  to  not  confuse  parameters  with 
measures.  If  it  can’t  be  expressed  as  a  probability 
it  probably  is  not  an  effectiveness  measure. 
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